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These tutorials are a simplified 
introduction, and are not sufficient on 


“Engineering is achieving function their own to achieve system safety. 
while avoiding failure.” You are responsible for the safety of 
; your system. 


— Henry Petroski © 2020 Philip Koopman 1] 


Carnegie 


Is Your System Appropriately Safe? one 
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= Anti-Patterns for Embedded System Safety: 


e Requirements do not address safety General Motors recalls 4 
million vehicles after software 
e Not using an appropriate safety standard linked to 1 death ws:rgo,uevee 


e Safety analysis assumes perfect Software =r ss te mrs seing nd digit 


eae perth ae ee and 
controls air bag elo nto test mode. If the a appens, the 


e Redundancy management inadequate Sonics no a Ft 





= Actually know system is safe 
e Correctness is only a starting point 
— Requirements and other aspects matter 
e Fault responses must be safe 
— Hardware faults (permanent; transient) 
— Software faults 
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Defense-In-Depth For Safety Mey 
aschmilgaion evelattempts —g_ Avoid faults occurring 
EEL e Careful design of software to avoid software defects 
a AVO 2 yt e Use robust hardware to avoid hardware run-time faults 
7 het [FAULT ra = Detect and contain faults 
DETECT & e Error correction HW, redundant CPUs 
' CONTAIN FAU LTS e Watchdog timers for failed tasks, exception handling 
| HAZARD m Use Fail Safe strategies to mitigate hazards 
7 *- BAIL SAFE e For example, automatic safety shutdown mechanisms 
ee ee * TENT = Incidents require operator intervention (or luck) 
Tre Serer e Operator may be able to react correctly and quickly 
reg aati sii e Incident will be a mishap some fraction of time 
(or, get lucky) = Want to avoid escalation as much as possible 


e E.g., fail safe approaches that work to avoid incidents 


| MISHAP | (For more information, see Safeware, Leveson 1986, pp. 149-150) © 2020 Philip Koopman 3 
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Basic Safety Principles wasraisss 


Safety must be seen to be present 
e System presumed unsafe unless convincing safety argument made 

e Outsider must be able to determine safety purely from documents 
The greater the risk, the greater the need for information 

e Riskier systems require more engineering rigor 

Safety must be built in, not added on 

e If code is created without a safety process, throw it away; start over 
Systematic, random, and malicious faults all matter 
e Consider design errors and transient faults (e.g., soft errors) 
e lf it's not secure, it’s not safe 

Safety must be argued in writing and demonstrated 

e Failure-free testing isn't enough 

Safety is a lifecycle concern 

e “Mission critical failures” can be considered “safety” as well 
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Safety Culture: Everyone Is Sure It's Safe eee 





=m Space Shuttle Challenger Mishap 
e January 1986 launch explosion; 7 fatalities 


e Dual O-rings keep hot gases inside solid booster 


—- History of sometimes failing if too cold 
— At launch, joint temperature was below freezing 
e Booster team told: “prove launch is unsafe” 


— Should have been: “no launch un/ess proven safe" 





— Getting lucky is not the same thing as being safe 
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Overview of Embedded System Safety aes 
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= Safety Topics: THERAC25 TREATMENT 
e Safety Plan & Safety Standards 





merssitch STABLE 














e Safety Requirements 
e Critical System Design | Ss 
e Dependability 
e Single Points of Failure | 
e Redundancy Management O34 
e Isolation Mechanisms co 
e Safety Architectural Patterns | 
(1985 - 1987) THERAC 25 
= Pitfall: Software-Controlled Radiation Therapy Mishaps 


e Safety isn't just about whether you think it's safe ... 
.. Is about whether you can prove it is appropriately safe 
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https://xkcd.com/1992/ 
MY CUBESAT PROPOSAL JAS THE FIRST TO BE REJECTED FOR 
VIOLATING EVERY DESIGN AND SAFETY REQUIREMENT SIMULTANEOUSLY. © 2020 Philip Koopman 7 


